Areas of UK for plotting#

The problem this part is trying to solve is to find a way to seperate addresses into larger regions:

To identify differences in regions
To be able to plot the data onto a map

The input data has a UK postcode, this can then either be converted to a latitude-longitude pair or a larger region that contains many post codes, such as a county or city region.

I’ll use the latter method so I can produce choropleth maps. There are a few different tools that can be used to plot these in python. I’ll use folium mainly because I have used it before.

To produce the choropleth maps from the postcode data, requires two things:

A way to transform postcodes to larger regions
Details of the polygons for those regions

Note: the regions in the two need to be the same.

For smaller regions the UK census regions work well or maybe the best way to go is with a nearest neighbour algorithm?

Code to create maps#

# geometry data
import fiona
import geopandas as gpd
import json

# general - working with dataframes and numbers
import pandas as pd
import numpy as np

import os

# plot map parts
import folium
import branca.colormap as cm

def split_islands(df):
    from shapely.geometry.polygon import Polygon
    from shapely.geometry.multipolygon import MultiPolygon
    
    # the islands is the 7th component
    gdf = df.iloc[7:8].copy()

    # slight mods to help change things
    gdf["geometry"] = [MultiPolygon([feature]) if isinstance(feature, Polygon)
                       else feature for feature in gdf["geometry"]]

    # explode th eislands into a number of polygons from one multipolygon
    gdf_parts = gdf.explode(column='geometry', ignore_index=True, index_parts=False)

    # take the first 3 elements only and dissolve back to one multi polygon
    df7new = gdf_parts.iloc[0:3].dissolve()

    # create the new geopanda with the new islands and the previous rest
    dfnew = pd.concat([df.loc[:6], df7new]).reset_index(drop=True)
    
    return dfnew

def kml_create_json(url_data, loc_save='./_data', fname='region.json', doScotSplit=False):
    # so can use kml files
    fiona.drvsupport.supported_drivers['KML'] = 'rw'

    # load the kl file to a gpd df
    df = gpd.read_file(url_data, driver=fiona.drvsupport.supported_drivers['KML'])

    if doScotSplit:
        df = split_islands(df)
        
    # Save as a json file to load in plot_map
    file_path = os.path.join(loc_save, fname)
    df.to_file( file_path, driver="GeoJSON")

    
    return df.Name, file_path

def plot_map(df,what_to_plot='amount_pc',region_to_plot='Name',
             json_path='./_data/region.json',
             longitude=-3.1, latitude=54.1):
    

    m = folium.Map(location=[latitude,longitude], 
                   zoom_start=5,
                   control_scale=True,
                   tiles="Stamen Toner")
    
    folium.TileLayer('CartoDB positron',name="Light Map",control=False).add_to(m)

    choropleth = folium.Choropleth(
        geo_data=json_path,
        name='choropleth',
        legend_name= what_to_plot,
        data= df,
        columns=[region_to_plot,what_to_plot],
        key_on= "feature.properties.Name",
        fill_color='YlGn',
    ).add_to(m)
    
    choropleth.geojson.add_child(
    folium.features.GeoJsonTooltip(['Name'],labels=False)
    )
    
    return m

def url_KML_map(url_data, doScotSplit=False):
    # create a json file for plotting and gives back names of regions
    fname=url_data.split('/')[-1].split('.')[0]+'.json'
    map_names, json_path = kml_create_json(url_data, fname=fname, doScotSplit=doScotSplit)

    df = pd.DataFrame(columns=['County','Data'])
      
    # add the names of the regions
    df['County'] = map_names
    # create some random data to plot
    df['Data']= np.random.randint(0 ,100,len(df) )

    m = plot_map(df,what_to_plot='Data',region_to_plot='County',
                json_path = json_path)
    
    return m

Create maps#

From UK postcodes, i.e. first two letter of the postcode like NE, M, PL, WA. This is the easiest way as the postcode input data can be easily transformed.

However, the regions are less intuitive. i.e. what is WA or PL? And some postcode areas can be very small or big such as London regions (small) or N. Ireland (big).

url_data = "https://www.doogal.co.uk/kml/UkPostcodes.kml"

url_KML_map(url_data)

Make this Notebook Trusted to load map: File -> Trust Notebook

For England#

Both the polygon file for counties and postcode to county conversion can be found at doogal

url_data = 'https://www.doogal.co.uk/kml/counties/Counties.kml'
url_KML_map(url_data)

Make this Notebook Trusted to load map: File -> Trust Notebook

For Scotland#

Postcode data can be found in numerous places (including Doogal), here the National Records of Scotland is used.

For the polygon data the UK Data Service is used.

The Scottish Parliamentary regions is used, with the area_to_plot='ScottishParliamentaryRegion2021Code column from the postcode data from National Records of Scotland.

Note slight difference in codes

The .kml file is very large mainly because the Highlands and Islands multi-polygon is so detailed to inorporate all the islands.

# url_data = "https://borders.ukdataservice.ac.uk/ukborders/easy_download/prebuilt/kml/Scotland_preg_2011.zip"
url_data = "C:\\Users\\44781\\Documents\\GitHub\\PesticideDocs\\_data\\scotland_preg_2011.KML"

url_KML_map(url_data,True)

Make this Notebook Trusted to load map: File -> Trust Notebook

A bit on Folium#

Part 1

m = folium.Map(location=[latitude,longitude], 
                   zoom_start=5,
                   control_scale=True,
                   tiles="Stamen Toner")
    
folium.TileLayer('CartoDB positron',name="Light Map",control=False).add_to(m)

Part 2

choropleth = folium.Choropleth(
    geo_data=json_path,
    name='choropleth',
    legend_name= what_to_plot,
    data= df,
    columns=[region_to_plot,what_to_plot],
    key_on= "feature.properties.Name",
    fill_color='YlGn',
).add_to(m)

Part 3

choropleth.geojson.add_child(
folium.features.GeoJsonTooltip(['Name'],labels=False)
)

Above is the function used to plot maps in this page using Folium

Create a basic map:
- This sets the basic look of the map
- things like the zoom, color scheme, where the map is centred etc
Create the Choropleth part:
- This creates the Choropleth map (i.e. regions on a map of different color based on their value)
- json_path this is the path to a json file with the location data
  - key_on= "feature.properties.Name" refers to the json file
  - and needs to match region_to_plot in columns=[region_to_plot,what_to_plot]
- data the data normally a pandas dataframe
  - the columns are region_to_plot and what_to_plot
  - what_to_plot are the values to plot
  - region_to_plot should match the json file
This just adds ability to see names of regions on mouseover

Going from postcode to area#

To create a function that takes in a postcode and outputs a region the following was done:

Load the postcode and region data to a dataframe
- i.e. pcode = pd.read_csv(os.path.join(dirname,filename), usecols=['Postcode',area_to_plot])
Convert it to a dict pcode_dict = dict(pcode.values)
Use map to convert the postcodes to area
- df[area_to_plot]=df.loc[:,'Postcode'].map(pcode_dict)

Defra Pesticide Testing, Data Analysis

Areas of UK for plotting

Contents

Areas of UK for plotting#

Code to create maps#

Create maps#

For England#

For Scotland#

A bit on Folium#

Going from postcode to area#